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(57) ABSTRACT 

Enhanced operation is attained for packet networks with a 
performance management operations system (PMOS) that 
receives information from network elements concerning 
loads carries and lost packets, receives threshold and other 
parameter information for a network management console, 
performs various calculations on the received information, 
and develops recommendations for setting of adjustable 
network elements controls that affect the quality of service 
that those elements provide. For those network elements that 
are capable of receiving operation control signals from the 
PMOS and that the network management console ceded a 
measure of control to the PMOS, the developed recommen- 
dations are converted to control signals that are directly 
applied by the PMOS to the network elements to control the 
network elements* operations. 

26 Claims, 1 Drawing Sheet 
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PACKET NETWORK PERFORMANCE BRIEF DESCRIPTION OF THE DRAWING 

MANAGEMENT ^ presents a block diagram of an arrangement where 

REFERENCE TO RELATED APPUCAHON a PMOS interacts with networks elements and with a 

console to control quality of service offered by the network; 

This application claims the benefit of provisional appH- ^ g^d 

cation No. 60/073,195, filed Jan^ 30 1998, tiUed "Data pj^ ^ ^ -^^^ functional components within the FIG. 

Structures and Algorithms for ATM Performance Manage- ^ PMOS 
ment OS." 


BACKGROUND OF THE INVENTION 10 


DETAILED DESCRIPTION 


i..r. 1.^1 J Before proceeding with the detailed description of this 

ms mvention re ates to fast packet networks and njore ^^^^^^ ? ^^J,^^, ^^^^ be gained from reading 

particularly, to performance management and control of fa„,iiiarizing oneself with the following texts: 

such networks. . , . [1] ATM Forum, "Traffic Management Specification, Ver- 

The voice network comprises a large plurality of inter- ^.^^ 4 ^ » Document number af-tm-00556.000, March 

connected sAvitches that are arranged hierarchically and 1996. 

adapted to route voice traffic. The notion of managing such ^^^^^ "Private Network-Network Interface 

a network is not new In the context of such voice networks. Specification, Version 1 .0," Document Number af-pnni- 

the network management paradigm includes having the 000 April 1996. 

switches report various status indications, such as what load pj p^^^^ «M4 Network Element View Interface 

they carry, and what spare capacity they have to a network Specification and Logical MIB, Version 2.0," Document 

management center. Based on this mformation. the network ^isymh^i af-nm-0020.001 (in publication), 

management center controls routmg of forthcommg caUs [^-j internet Engineering Task Force, "Management Infor- 

when necessary (e.g. when switch failures occur). ^^^^^^ Network Management of TCP/IP-based 

Packet networks (e.g. ATM, IP, and Frame Relay) are 25 Internets: MIB II," Document number RFC- 1213, March 

quite different in their operation and consequent behavior 1991. 

and, till now, there has not been an overall network man- [5] internet Engineering Task Force, "IP Forwarding Table 
agement operations system that could be applied to such MIB,'* Document number RFC- 1354, July 1992. 
packet networks. Companies like Cisco and NewBridge do piG. 1 depicts an ATM PMOS 100 that comprises a 
make element management systems (EMSs) which interact 30 processor 110 coupled to memory 120. Illustratively, ATM 
with a plurality of their respective network elements (e.g., network elements (NEs) 210, 220, 230 are coupled to PMOS 
routers and switches). An interface has been standardized for 100 ^nd interface with PMOS 100 in accordance with an 
the communication of information between network ele- agreed-upon interface protocol, such as the M4 ATM inter- 
ments (NEs) and the EMS with which they communicate. face standard, or an enhanced version thereof. Also coupled 
Different types of networks have different standards, how- 35 to PMOS 100 is an Element Management System (EMS) 
ever. For example, the standard for ATM networks is dff- 240 which, in turn, is coupled to a number of NEs. EMS 240 
ferent from the standard for IP networks. a system of the prior art variety that receives information 
To provide a view of the entirety of a packet network, from a number of NEs and reports that information; in this 
there is a need to combine the information that is obtained case to PMOS 100. Lastly, FIG. 1 includes a network 
from the numerous EMSs that are installed in the network, 40 management console 300 that is coupled to PMOS 100. An 
and some of the above-mentioned companies have operator responsible for the network interacts with PMOS 
attempted to create such systems. However, the available 100 through console 300. 

systems appear to only report on the provided information FIG. 2 illustrates some of the functional components of 

and develop alarm reports when thresholds are exceeded, but PMOS 100. The primary mission of performance data 

donootheranalysisof the data. Consequently, these systems 45 collector 130 is to collect the raw data from the NEs and 

cannot make recommendations for altering the behavior of distribute it to the other units. Although this can be done with 

the NEs and, expectedly, do not have a mechanism for microprocessor 110, it can also be done with a separate 

actually controUing the NEs. This, of course, leads to a less adjunct, such as Concord Network's Health™ system. Some 

than optimized utilization of the packet network. of these commercial collectors also performs some analysis 

50 of the data and provide a report that reflects this analysis. For 
SUMMARY example, some of the commercial collectors can identify the 
The problems of the prior art are overcome, and enhanced ten worst performing ports, 
operation is attained for packet networks with a performance The output of performance data collector 130 is fed to 
management operations system (PMOS) that receives infer- capacity manager 140, to congestion manager 150, to Ser- 
mation from network elements concerning loads carried and 55 vice Level Agreement reporter 160 and to tuner 170. Capac- 
lost packets, receives threshold and other parameter infor- ity manager 140 gathers the information provided by col- 
mation for a network management console, performs vari- lector 130 and maintains a historical store of the data 
ous calculations on the received information, and develops supplied by collector 130. With the aid of various generated 
recommendations for setting of adjustable network elements reports that are sent to console 300 upon request of the 
controls that affect the quality of service that those elements 60 console, capacity manager 140 guides network engineering, 
provide. For those network elements that are capable of such as installation of new capacity, etc. Congestion man- 
receiving operation control signals from the PMOS and that ager 150 provides reports to console 300 regarding conges- 
the network management console ceded a measure of con- tion problems that need immediate, or near real time, 
trol to the PMOS, the developed recommendations are attention. For example, a radio station that broadcasts some 
converted to control signals that are directly applied by the 65 contest can expect a large volume of traffic, and such traffic 
PMOS to the network elements to control the network can easily cause an overload. Congestion manager 150 
elements' operations, report to console 300 can alert the console operator of such 
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a congestion condition to allow the operator to perform 
whatever controls that the operator can exercise over and 
above what tuner 170 does. Congestion manager also reports 
on conditions such as non-availability of ports or other 
equipment modules, loss of data, and failures to properly 
collect data. Service Level Agreement reporter 160 is 
charged with formulating reports to large customers regard- 
ing the overall service that the customer is receiving over the 
entire network. For example, there are many enterprises 
such as Sears, General Motors, etc. that contract for sub- 
stantial network resources, and it is the mission of reporter 
160 to create reports that would inform such customers 
whether the service they do receive meets the agreed-upon 
and contracted service levels. 

The accent of this disclosure regards tuner 170, which 
based on the inputs provided by collector 130, recommends 
settings for the adjustable controls of the NEs and, where 
possible, actually affect those controls. 

Functionally, all of the NEs are the same in the sense that 
they carry packet trafiSc, However, there is no requirement in 
the FIG. 1 arrangement for the NEs to be of any one given 
variety or manufacture. Different types of NEs are perfectly 
acceptable. Still, each NE includes a plurality of input ports 
on which incoming packets are received, and a plurality of 
output ports on which outgoing packets are delivered. Cus- 
tomer devices (not shown) are coupled to some of the ports. 
When a customer device wishes to place a call, it negotiates 
a particular level of service with the network, and when the 
negotiation is successful, communication commences. 
There are a number of different levels of service that can be 
negotiated, ranging from the poorest Quality-of-Service 
(QoS) that guarantees no particular rate of accepted packets, 
to the highest QoS that guarantees a given, constant, bit rate. 

When the network engages in the negotiations and guar- 
antees a certain level of service to its customers, it concerns 
itself with the load to which the ports in each NE in the 
network commit themselves to carry. Obviously, it is proper 
for an NE to commit one of its ports to carry its fiill capacity, 
and one can assume that such a condition does occur from 
time to time. Even when such a condition occurs, however, 
it still remains that not all customers continuously transmit 
at the maximum agreed-to rate and, therefore, some capacity 
is typically left unused even at a fully committed port. This 
capacity can be used by active customers to send packets at 
a rate higher than agreed-to, although this higher rate is not 
guaranteed. Also, the network can afford to overbook 
somewhat, or over-commit, some ports of an NE, with 
reasonable assurance that the ports will still meet their QoS 
agreements with its customers. Of course, in the case of 
capacity overbooking there is a finite chance that packets 
received at a network element will find themselves blocked 
because the output port through which they need to leave is 
fully occupied. 

While the above-described mechanism that contributes to 
packet loss is the most prevalent cause of packet loss and is, 
therefore, the most important, it should be realized that 
incoming packets may also be lost within a network element 
because the header information of the packets is somehow 
corrupted or is otherwise inappropriate, or incorrect. While 
such losses are not necessarily related to overbooking, they 
do affect QoS and, therefore, some embodiments of PMOS 
100 concern themselves with such losses. 

Associated with the task of negotiating the admission of 
connections, the NEs of the FIG. 1 network have adjustable 
parameters that contribute to the decisions on connection 
admission, overbooking, etc. The purpose of these adjust- 
able parameters is to permit network operators to achieve a 
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satisfactory level of network performance under heavy loads 
of the hard-to-model packet traflBc. The tuning of these 
adjustable parameters has proven to be difiBcuIt, particularly 
when this is attempted on the basis of data collected from 

5 only the particular NE to be tuned, or from a few NEs. It is 
anticipated that such difficulty would be compounded in an 
arrangement where different types of NEs are employed that 
are manufactured by one manufacturer or a number of 
different manufactures. It may be noted that different types 

10 of NEs can have different capabilities in connection with the 
adjustable parameters. For example, some may have the 
ability to affect the overbooking factor on a port basis, while 
other can affect the overbooking factor only for sets of ports. 
In accordance with the principles disclosed herein, the 

15 adjustable parameters of the NEs are controlled based on 
information that is collected and analyzed from the entirety 
of the network, and not just from the controlled NE. 
Accordingly, each of the ATM NEs periodically sends 
information to PMOS 100, either directly or through an 

20 EMS. 

Illustratively, the anangement of FIG. 1 corresponds to an 
ATM network. The discussion that follows relates to ATM 
networks, and in the context of ATM networks, packets are 
also called cells. In the discussion that follows the term 
25 "cells" is often used, but it should be clearly understood that 
the principles disclosed herein apply to all packet networks 
and not merely to ATM networks. 

Thus, at least with the granularity of reporting on each 
port of the NE, the NE sends information that, illustratively, 
30 includes: 

Cells received. 

Cells transmitted. 

Cells lost due to congestion, 

Cells lost due to header error check violations and other 

protocol errors 
Utilization factor. 
Spare capacity, and 

Values of tunable parameters that affects loss of cells. 

40 Other data can also be included, of course, such as reporting 
on status of data links and related equipment. Also, perhaps 
off-line, other data is provided to inform PMOS 100 of the 
network topology, the connectivity of the NEs, the capacity 
of each link in the network, the capacities already committed 

45 to the various virtual connections, the current settings of the 
adjustable parameters, etc. This information is mostly used 
by capacity manager 140. 

It may be noted that the general behavior of the network 
does not change dramatically from one instant to the next 

50 and, therefore, the sending of information from the NEs to 
PMOS 100 need not be continuous. We selected a nominal 
data collection interval of 15 minutes, and in those NEs that 
employ this interval, transmission of data from each NE to 
PMOS 100 occurs every 15 minutes. It should be 

55 appreciated, however, that not all network elements have to 
have, or indeed should have, the same data collection 
interval. It is quite possible to have some types of network 
elements have a shorter or longer data collection interval, 
and the distinction need not necessarily be as to type. It 

60 could, for example, be related to the volatility of load 
changes that a particular NE has historically experienced. 
Accordingly, PMOS 100 provides a signal to the NEs that 
synchronizes the reporting of data, and specifies the data 
collection intervals. 

65 It may be observed that the connection between PMOS 
100 and NE 230 is unidirectional — only from NE 230 to 
PMOS 100. This intends to show that PMOS 100 is not 
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limited to one type of NE, is not limited to NEs that are of 
a particular manufacturer, and is not limited to interacting 
only with NEs that can communicate bi-directionally. The 
uni-directional operation relates to the ability to specify 
collection intervals, and synchronization of the reporting. It 5 
does not necessarily relate to the ability of PMOS 100 to 
control the adjustable parameters. It is possible for the FIG. 
1 network to work with NEs that do not admit of any 
electronic control of parameters, although it is recognized 
that at least in connection with such NEs, optimized control 
is not likely to occur. 

As depicted, PMOS 100 comprises a single processor 110. 
It should be understood, however, that processor 110 can 
comprise a plurality of sub-processors, for example, micro- 
processors. Different sub-processors can easily handle dif- 
ferent NEs that report asynchronously (such as NE 230). 
Also, being aware that processors are available which can 
handle a number of asynchronous interrupts, one can appre- 
ciate that receiving reporting data from effectively asynchro- 
nous NEs such as NE 230 is quite easy, in spite of the fact 
that communication is uni-directional. Of course, for NEs 
that accept synchronization signals, it is expected that the 
collection intervals would be synchronized, at least so that 
reports can be made with reference to some common time 
interval, such as "busy hour." ^5 

It is expected that at least in some embodiments of this 
invention, the specifics about the reporting NEs, such as 
type, and NE identifier, are known to PMOS 100 a-priori by 
virtue of the manner by which the NE makes connection to 
PMOS 100, time of reporting, etc. In other words, the 
connections between PMOS 100 and the NEs in such 
embodiments are provisioned and administered a-priori. 
Other embodiments, however, allow for each transmission 
from an NE to be accompanied by information that supplies 
the NE type, an NE identifier, an NE port ID, and the time 
interval that is represented by the reported data. 

The reporting of information from the ATM NEs may be 
less granular than merely reporting on cells of a port. 
Specifically, the periodic data reported may include infor- 
mation for each provisioned Mrtual Channel Link (VCL) 
and Virtual Path Link (VPL), such as 

Cells received at each selected VCL or VPL. 

Cells transmitted at each selected VCL or VPL. 

Cells discarded at each selected VCL or VPL due to either 
a Usage Parameter Control (UPC) or Network Param- 45 
eter Control (NPQ — in the receive direction. 

High priority cells discarded at each selected VCL or VPL 
due to UPC/NPC action — ^in the receive direction. 
A VPL comprises a collection, or bundle, of VCLs. Usage 
Parameter Control is a control that is exercised at the point 50 
where a source of packets connects to the ATM network. 
When a call is negotiated and provisioned, a certain maxi- 
mum packet rate is agreed upon. When such a source 
transmits at a rate higher than agreed on, and there is not 
enough capacity in the network, the UPC throttles the 55 
source. NPC is similar to UPC, except that this control is 
exercised at the coupling between the ATM network of one 
provider and the ATM network of another provider (because 
each provider worries only about not overloading its own 
network). 60 

Not all VCLs and VPLs on a given ATM NE are generally 
reported on. Rather, a maximum number of VCLs and VPLs 
is set, and that number generally varies with the type of the 
subject NEs. The VCLs and VPLs that are candidates for 
such monitoring are estabhshed through provisioning. The 65 
VCLs and VPLs belonging to all Permanent Virtual Con- 
nections (PVCs) are candidates for such selection. Depend- 


ing on the type of ATM NE, the VCLs and VPLs that are 
associated with Soft PVCs (SPVCs) can also be candidates 
for such provisioning. 

PMOS 100 provides a control signal to direct an NE to 
initiate or terminate the reporting about VCLs and VPLs. 

With respect to a particular NE, a given call that is 
handled by the NE is denoted by its Virtual Call Connection 
(VCC), for example VCC-100. Denoting a VCL in an 
incoming link that supports VCC-100 by VCL^, and denot- 
ing a VCL in an outgoing link that supports VCC-100 by 
VCL^, we define a Cell Loss Ratio (CLR) in a given 
direction by: cells arriving at VCL^, less cells arriving at 
VCL^, divided by cells arriving at VCL^. The numerator is, 
of course, a measure of the cells lost. In other words. 


CZJf = 


Cells lost to veil, 
Cells transmitted via VCLi, 


To generalize. Packet Loss Ratio, or PLR, is defined by 

Packets lost 


PLR = 


Packets transmitted' 


This definition holds whether it is for a VCL, a VPL, or a 
port. Of course, when a port carries traflSc of different classes 
(discussed in more detail below) packets lost in VCLs that 
carry traffic with no promised Quality of Service are not 
considered in evaluating PLR because there is no reason to 
degrade the effective PLR of a port due to lost packets in 
support of a service that requires no particular level of QoS. 

To effectively process the periodically collected trafiSc 
and performance data already described, additional infor- 
mation is needed by PMOS 100 concerning (1) traffic 
descriptors for VCCs and Virtural Path Connections (VPCs), 
(2) the correlation between VCCsA^PCs and VCLs/VPLs, 
and (3) the correlation between VCLsATLs and the ATM 
NE ports. Accordingly, this information is included in the 
reporting to PMOS 100, including the traffic direction to 
which the data pertains. 

With respect to traffic descriptors, illustratively, the fol- 
lowing information is provided and associated with a par- 
ticular ATM NE port and traffic direction. 

Provisioned VPC Traffic Descriptor Inforaiation 

Virtual Path Link identifier. 

Peak Cell Rate of Virtual Path Connection (PCR-VPC). 

Provisioned VCC Traffic Descriptor Information 

Virtual Channel Link identifier. 
Service category. 

Peak Cell Rate of Virtual Channel Connection (PCR- 
VCC). 

Sustainable Cell Rate of Virtual Channel Connection 
(SCR-VCC). 

Maximum Burst Size of Virtual Channel Connection 
(MBS-VCC). 

Minimum Cell Rate of Virtual Channel Connection 
(MCR-VCC) for ABR service only. 

Threshold crossing alerts is one of the measures that 
PMOS 100 develops, either directly from received data 
that reports about such alerts (such as from EMS 240) 
or from the raw data that PMOS receives. 

Cells lost due to congestion on each AI^ NE port (i.e. in 
the transmit direction). 
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Ctell lost due to Header Error Check violations and ATM 
protocol error on each ATM port (i.e. in the receive 
direction). 

Cells discarded at each selected VCL of VPL due to of 
Virtual Channel Connection (PCR-VCC) High priority 5 
cells discarded at each selected VCL of VPL due to of 
Virtual Channel Connection (PCR-VCC). 
Advertised available bandwidth per port, per service 
category, as provided by the previously referenced 
Private Network-Network Interface (PNNI) standard. 
The following describes some of the basic analyses that 
PMOS 100 carries out, which are then employed to develop 
recommendations for altering behavior of the network ele- 
ments and for effecting such recommendations. 

15 

Busy Period Determination 
Prior to identifying the "busy period," a number of 
parameters need to be selected in order to have a meaningful 
measure. 

1. Duration of the busy period (category-1 interval). PMOS 20 
100 employs a default duration of one hour as the as 
duration of a busy period, synchronized to the "wall 
clock." However, PMOS 100 accepts inputs from console 
300 for any other operator-specified reference that is 
consistent throughout the network. Advantageously, such 25 
duration would be a multiple of the measurement and 
reporting interval (category-0 time interval). 

2. Interval of time within which the busy period is selected 
(category-2 time interval, corresponding typically to num- 
ber of category-1 time intervals). A default time of 24 30 
hours seems appropriate but, of course, the operator of 
console 300 can make any other choice. 

3. Selection of specific dates to be included in the database 
for the busy period determination (category-3 time 
interval, corresponding to a number of category-2 time 35 
intervals). In our embodiment, we employ a default 
interval of 20 days. As in category-1 and category-2 time 
intervals, above, the operator of console 300 can make 
any other choice. 

4. Choice of either (a) time-consistent period (which is the 40 
default) or (b) extreme -value engineering busy period, 
such as "bouncing busy hour". Based on this parameter, 
one can have a measure of traflSc conditions, for example, 
within any selected hour of the day of the last 20 days 
("time consistent**), or the measure of traflSc at the busiest 45 
hour in each day of the last 20 days ("bouncing busy 
hour*'). 

The actual busy period determination is quite 
straightforward, since the raw data can easily be organized 
by PMOS 100 in a table format that includes, for example, 50 
a column for each of the following: NE Type, NE ID, NE 
port ID, time interval (e.g. 10:00 EST, 10:15 EST, etc.), 
Date, VCC ID, VCL ID, Cells received. Cells transmitted. 
Cells lost due to congestion, Cells lost due to HEC or ATM 
protocol error, service category, etc. Selecting an array from 55 
this table that corresponds to the category-3 time interval 
(for example, 20 days) allows one to identify whatever 
sub-array of data is necessary (for example, the sub-array of 
data for a particular VCC of a particular port) and identify 
the busiest category-1 time interval (for example, one hour) 60 
in each of the category-2 time intervals (for example, in each 
day). 

The above describes, basically, a database query and 
selection operations. Of course, the data can be stored in a 
bona fide commercially developed database, and the formal 65 
database language that comes with the selected database 
product can be employed. By way of example, we consid- 
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ered using Oracle and Informix products, which are database 
products that are geared to large databases. 

The above demonstrates that busy period determinations 
can be made with fairly fine granularity. Of course, the 
determinations can be made with less fine granularity by 
simply combining information, or selecting information in a 
different manner. For example, one can easily select the data 
relative to the entirety of an ATM NE rather that selecting for 
a particular VCC. On the other hand, some additional 
information is derived when PMOS 100 analyzes the fine- 
granularity data and develops measures such as mean, 
median, standard deviation, etc. While such data might not 
be of immediate value in the control of the adjustable 
parameters of the network NEs, reporting of this data to 
console 300 is of assistance in the future engineering of the 
network. 

Busy Period Load 

PMOS 100 provides busy period load information. 
For a Time Consistent Busy Period determination, PMOS 
100 provides the sample mean over the category-3 time 
interval. For example, PMOS 100 may determine that, on 
the average, a given port transmitted X cells between 10:00 
AM and 11:00 AM over the past 20 days. PMOS 100 also 
provides the average busy period utilization factor of ports, 
which corresponds to the average load of the port under 
consideration, divided by the capacity of the port (where 
capacity of the port corresponds to the Peak Cell Rate (PCR) 
of such a port). That is, the average utilization factor, p, is: 

^ count of transmitted cells at NE Port during interval 

'mterv(ds 

^~ PC/f(po«)x total lime of intervals 

Applications of Bouncing Busy Period data often involve 
the use of both the sample mean and the sample variance of 
the load in order to support Extreme Value Engineering 
methods. Accordingly, for a Bouncing Busy Period deter- 
mination based upon a category-3 time interval specified by 
the operator of console 300, PMOS 100 provides the sample 
mean and the sample variance (over the set of category-2 
time intervals), and the average busy period utilization 
factor. 

Busy Period CLR Performance — Congestion 

The busy period congestion-related CLR performance for 
a selected set of ATM NE ports and for a selected category-3 
time interval can be found from the busy period determina- 
tion by simply culling out the appropriate information from 
the table; specifically from the "Cells received" column the 
"Cells lost due to congestion" column. Thus, PMOS 100 
provides, upon request from the operator of console 300, any 
or all of the following: 

1. The sample mean (over the busy periods for a specified 
category-3 time interval) of the cells lost due to conges- 
tion. 

2. The sample mean (over the busy periods a specified 
category-3 time interval) of the cells transmitted, i.e., the 
sample mean of the busy period load. 

3. The ratio of item 1 to item 2, i.e., the estimated CLR. 

4. The average busy period utilization factor. 
The relative busy period congestion-related CLR perfor- 
mance for each of the ATM NE Ports associated with a 
specified category-3 time interval can be found from the 
busy period as determined above and information for each of 
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these ATM NE Ports of the type shown by the "Time 
interval", "Cell transmitted", and "Cells lost due to conges- 
tion" columns of the table. 

Additionally, PMOS 100 provides any or all of the 
following information upon request from the operator of 
console 300: 

1. The sample mean (over the busy periods for a specified 
category-3 time interval) of the cells lost due to conges- 
lion on each ATM NE Port. 

2. The sample mean (over the busy periods for a specified 
category-3 time interval) of the cells transmitted on each 
ATM NE Port, i.e., the sample mean of the busy period 
load. 

3. The ratio of item 1 to item 2 for each ATM NE port, i.e., 
the estimated CLR. 

4. The average busy period utilization factor. 
Associated with these measurements, PMOS 100 identi- 
fies the interval for which the computations relate, the busy 
period (if a Bouncing Busy Period is used), and a list of all 
ATM NE ports that are included, ranked from the highest 
CLR to the lowest CLR. As indicated above, PMOS 100 
accepts threshold inputs fi^om console 300, and in response 
to a request, provides the above-identified information only 
for those ATM NEs, or only for those ports, which exceed 
the thresholds. This provides an ad hoc mechanism for 
reducing the amount of data that is sent to PMOS 100. 

Busy Period CLR Performance — Header Error and 
ATM Protocol Processing 

The above-described process focuses on congestion- 
related CLR performance, and it is anticipated that the 
performance information derived from this process would be 
of significant value to managers of ATM networks. PMOS 
100 also provides information that focuses on CLR perfor- 
mance related to header error and ATM protocol processing. 
This information is derivable from the above described 
table, and more specifically, from the "Cells lost due to HEC 
or ATM protocol error** column, in conjunction with the 
table columns identified in connection with the "Busy Period 
CLR Performance — Congestion" section above. It should be 
realized that its is quite easy to substitute the "Cells 
received" column for the "Cells transmitted" column in the 
above-disclosed determinations. 

CLR Performance Processing for Other Periods 

The preceding focused on CLR performance processing 
for busy periods, which is anticipated to be the primary area 
of interest for network managers. It is anticipated, however, 
that CLR performance at other periods may be of interest. 
Accordingly, PMOS 100 provides all of the above-disclosed 
determinations for any time interval specified by the opera- 
tor of console 300. 

Class of Service 

The above discussion does not address class of service to 
make it clear that the above-described determinations can be 
easily employed in embodiments that comprise older ATM 
NE types which support only a single class of service. New 
types of network elements, however, support a plurahty of 
service categories. This includes a class of service which 
guarantees that the source would be able to send packets at 
a constant bit rate (CBR). A lower class of service allows 
real-time variable bit rate. At this class of service, the 
network guarantees to deliver an agreed-upon average 
packet rate with a delay that is not greater than some level, 
in support of services that require real-time performance. 
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such as voice communication. A still lower class of service 
allows non-real time variable bit rate, which is the same as 
the real-time guarantee in terms of promised delivery of 
packets but with a much more lenient delay requirement. A 
5 yet lower class of service is known as Available Bit Rate, or 
ABR, service, which employs feedback to offer a bit rate that 
relates to the available capacity. Lastly, the lowest class of 
service is Unspecified Bit Rate (UBR), which guarantees no 
particular QoS. 

10 When the ATM network works only with a single class of 
service, and that class is the CBR class, the allocations at the 
NEs effectively correspond to "peak rate allocation," the 
exclusion of statistical muUiplexing and the exclusion of 
"overbooking." Under such operating conditions, the CLR is 

15 expected to negligible, except under operational error of 
assigning too much traflSc to a given ATM NE port. 
However, performance objectives for Cell Delay Variation 
(CDV) can impose loading limits for a given ATM NE port. 
CDV can be estimated using in-service performance moni- 

20 toring techniques. See ITU-T Recommendation 1.610, 
"B-ISDN Operations and Maintenance Principles and 
Functions," November 1995. 

CDV can be controlled by properly loading ATM NE 
ports. For a given ATM NE port, both its utilization factor 
and the total number of active VCCs on that port are 
relevant. See J. W. Roberts and J. T. Virtamo, "The Super- 
position of Periodic Cell Arrival Streams in an ATM 
Multiplexer," IEEE Transactions on Communications, Feb- 
ruary 1991, and C. A. Cooper, "Some Considerations for 
Cell Delay Variation," Standards Contribution, Document 
Number TlAl.3/94-096, October 1994. A simple algorithm 
for specifying the loading of any ATM NE port with a 
capacity of DS3 or higher and VCCs with a capacity of at 
least a DSO (equivalent to a PCR of about 170 cells/second) 
in a manner that ensures compliance with reasonable per-NE 
CDV objectives is: 

For CBR traflBc, this can be written in terms of the Peak 
CeU Rates of this Port and its VCCs: 

^ PCRiVCQ & 0.85 X PCRiPori) 
vcc 

45 

While this algorithm always yields safe results, it can be 
overly conservative — particularly when a port carries only a 
few VCCs having large capacities. More precise algorithms 
can be developed, for example by applying suitable pro- 

50 grams to an established method. See the aforementioned 
ITU-T Recommendation 1.610. 

It is noted that because this single service has a stringent 
CLR objective of about 10"^° per NE (i.e. one lost cell in 
10^° transmitted ceUs) considerable aggregation over a 

55 number of busy periods and/or a number of ATM NE Ports 
will generally be needed to verify objective conformance. 

The following considers embodiments where a nimiber of 
different classes of service are handled; illustratively, CBR 
and non-real-lime Variable Bit Rate (nrtVBR). 

60 The periodically collected trafl&c and performance data for 
ATM NE Ports is anticipated to be generally available. The 
periodically collected traflBc and performance data for 
selected provisioned VCLs and VPLs might not be routinely 
available from some ATM NE types, for sufiBcient quantities 

65 of provisioned VCLs and VPLs to meet the criteria 
described below. Actually, use of periodically collected 
trafiSc and performance data for selected provisioned VCLs 
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may place excessive loads on the data collection capabilities 9) Determine Busy Period CLR Performance — ATM Pro- 
of PMOS 100, and require the use of a large data base (for tocol Processing, with modifications corresponding to 
example, on the order of a Terabyte, for networks of those of item 8 above. 

reasonable size). There are several approaches for dealing 10) Determine CLR Performance Processing for other 

with these problems. First, the coUection of traffic and 5 Periods, with modification corresponding to those of item 

performance data on a per-Service Class basis would pro- 8 above. The impacted steps in this determination are the 

vide both a reduction in data collection loads and PMOS sample mean calculations, the interpretation of estimated 

database size. However, this approach to data collection is CLRs, and the utilization factor calculations, 

not yet standardized in the industry. Second, an operator- In the above determinations, it is assumed that the CLR is 

administered selection of VCLs that belong to particular 10 negligible for VCCs in the CBR service class, so that all 

VPLs, by Service Class, would permit data collections for a measured cell losses are attributed to VCCs in the nrtVBR 

small number of VPLs (rather than a substantially larger service class. While this assumption is usually safe, a user 

number of VLCs). may sometimes require a more careful check of the CLR as 

measured for WCCs on one or more ATM NE Ports in the 
Two Service Classes Without Periodically Collected 15 cBR service class. PMOS 100 provides this more careful 
VCL and VPL Data check using the process described below, which assumes 
The foUowing procedure is carried out by PMOS 100 to t^at the involved ATM NE types can provide periodically 
treat two service classes when the analysis proceeds without collected data on a sufBcient number of VCLs. 
the use of periodically collected data for all of the provi- 
sioned VCLs and VPLs on the ATM NE Ports under 20 Two Service Classes With Periodically Collected 
consideration. This procedure operates on an operator- VCL and VPL Data 

specified set of ATM NE Ports and with respect to an t-u ^ n • j • j * * * 

opvvixivu^ow^ v^j-^^^^ ^ ^^^^ interval foUowmg procedure is used to treat two service 

operator-speci e ca egory- ™^ J" • classes when the analysis can proceed with use of periodi- 

1) Idenuty the data that^avaalable with reference to the ^^^^^^^ p^visioned VCLs and 
specified se of ATM NE Ports to be considered VPLs on the ATM NE Ports under consideration. Ihis 

2) Determine the busy period and the busy penod load for ^^^^^^ ^p^^^,^^ ^ operator-specified set of ATM NE 
both service classes. ^ ^ ... Ports and an operator-specified set of Dates. 

3) Determine the capacity of each ATM NE Port associated . ■ ,. . c^. c .u A-m^ kte n . . u 
with the data ) Establish an array of data for the ATM NE Ports to be 

4) Determine the VCL identifiers for aU of the VCLs on each 30 consWered. ... 
a™ NE Port associated with the daU. 2) Determine busy period and overaU busy period load for 

5) Determine the Traffic Descriptor for each VCL found in ^"^y (^f ' ^^^'^ „ , . . , 
item 4. (Recall that the Traffic Descriptor includes iden- 3) Determme the capacity of each ATM NE Port associated 

tification of the service class ) ^ ^^^^ 

z-x j^. ' „r *u ATkJi KTc 35 4) Determine the VCL identifiers for all of the VCLs on each 

Sc bvT ^«"P'"="y°^^'"=''^'^NEPortfornrtVBR > ^^^^^ ^.^^ ^^^^ ^^^^^ 

5) Determine the Traffic Descriptor for each VCL found in 
PCR(Port-nrtVBR)=PRC(port)-PRC(poit-CBR), where item 4. (Recall that the Traffic Descriptor includes iden- 
tification of the service class via the ATM Fomm service 

40 category.) 

6) Determine the capacity of each ATM NE Port for CBR 


PRC{VCC~CBR) 

VCC^CBR 


PFQPor,-CB,D = • traffic by 

7) For each time interval and each ATM NE Port associated 2 PRC[VCC- CBR) 

with the data and the category 3 time interval, estimate the PCR{Porr - cbr) ^ 

count of cells transmitted for nrtVBR traffic by: 


0.85 


"Estimate of Cells Transmitted (Port-nrtVBR)"«»A-B and for nrtVBR traffic by 

Where A is the "Count of Cells Transmitted (Port)" and 50 PCR(Port-nrtVBR).PCR(Port-CBR) 
B is the "Estimate of Cells Transmitted (Port-CBR)" and 

is equal to 7) por each time interval and each ATM NE Port associated 

with the subject data and time interval, determine the 
Pffavcc-CBffjx Interval duration count of cells losl due to congestion for CBR traffic by 


VCC-CBR 55 


Count of Cells Losi( Porr - C5/f) = 

8) Determine Busy Period CLR Performance — Congestion, y q^^^ ^^Uj Lost( vcc - CB/f), 

with the following modifications: vcc^-vbr 

a) When computing the sample mean of the cells 

transmitted, use the result of above item 7 in place of , . . . , ^ « 

count of transmitted cells per port. summation is taken over aU VO^ on that Port 

b) Interpret the estimated CLR as related to the nrtVBR wbch support CBR service, a^^^^^^ 

^ " , lost due to congestion for nrtVBR traffic by 

service class. * ^ 

c) When computing the average busy period utilization 65 

factor, use the result of ilem 6 [i.e., PCR(Port- Count of Cells LosK/'orr- nnV^B/O^ 
nrtVBR)] in place of PCR(Porl). 
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-continued interval between transmission of control signals. As an 

V Coun, Cells Los« vcc-mtvc/?), f^pe" to NEs thai do not accept electronicaUy 

vcc^-vBR communicated control signals, the respective recommenda- 
tions are send to console 300 in the same manner as in 

5 embodiments where no NEs accept electronically commu- 

where the summation is taken over all VCLs on that Port nicated control signals. For all embodiments, however, 

which support nrtVBR service. PMOS 100 provides console 300 with a report thai details 

8) For each time interval and each ATM NE Port associated the history of the actual and recommended adjustable 
with the subject data and catergory-3 time interval, deter- parameter setting for the NEs. This report is provided on 
mine the count of cells transmitted for CBR trafiSc by either a scheduled basis, or upon demand. This report is 

useful for network design purposes. 

c ^ ,i T ^oD^ One of the analyses is, of course, whether any alarm 

thresholds have been exceeded. If they have, mformation is 

^ Count Cells TransX^CC - CBR), forwarded to console 300 to inform the operator of this fact. 

VCC-CBR A report that indicates which alarms were exceeded is more 

of an advisory, after the fact, type of output, which is not too 

where the summation is taken over aU VCLs on that Port «"isfactory asa means for controlling networkperfonnance. 

which support CBR service, and simQarly Determine the iJf """^ imPfant, output that is developed by 

count of ceUs transmitted for nrtVBR trafiSc by "^OS 100 is a control of the Connection Admission Control 

(CAC) function withm the ATM NEs. The function of the 
CAC algorithm within each ATM switch is to regulate the 

Count of Cells Transmn - nrtVBR) = number of ATM connections that are established through the 

y Count Cells Trans.iVcc-nrtvcR), NE, in Order to protect the level of performance 

vcc-nn-vBR expcrienced by those AI^ connections. 

Basically, given the busy hour load determinations, a peak 

. . • * 1 u ^ CLR is developed for each port of the NEs and Compares lo 

where the summation is taken over all VCLs on that Port . u- ^-i n wu j . ^ c i- «u 

... , .xmn • the objective CLR, which corresponds to the QoS of the 

which support nrtVBR service. ^ i , , ■ * ■ • j *u * «rt. 

9) Detertiine Busy Period CLR Performance-Congestion. "^^^ stnngent service that ^ earned on the port^ When the 

•*u r !i • j-c *' companson mdicates that the existing CLR is higher than 

with the foUowing modifications: ^^.^^^^^ ^^^^^^^^^^ ^^^^ 

a) When computing the sample mean of the cells lost due objective, it is clear that the CAC function must be made 
to congestion, use the result of above item 7 for CBR ^^^^ stringent so as to reduce the overbooking. If, on the 
service and for nrtVBR service in place of the count of ^^^^^ ^^^^ the peak CLR is lower than the objective CLR, 
cells lost due to congestion per port. (There are two ^ ^^^^^ ^hat one might be able lo increase the overbooking 
values to be reported.). somewhat and still remain within the given CLR objective. 

b) When computing the sample mean of the cells The amoimt by which the CAC is modified, however, is a 
transmitted, use the result of above item 8 for CBR function of the NE type, the utilization factor, and the current 
service and for nrtVBR service in place of count of capacity. The relationship among these factors is derived 
transmitted cells per port, (There are two values to be from a historical view of the data that is available in PMOS 
reported.) 100. This relationship is brought to light by selecting, from 

c) When computing the estimated CLR, there are two the corpus of historical data, a subset of data and basing the 
values to be reported — one for CBR service and one for CAC on the selected data. When the data is a database, an 
nrtVBR service. appropriate SQL SELECT statement might be: 

d) When computing the average busy period utilization SELECT CAC 
factor, use the result of above item 6 for CBR service 45 FROM [data corpus] 

and for nrtVBR service. Also of interest is the com- WHERE NE_Type-[type of NE under consideration], 

posite utilization factor, resulting in three' values of CLR"{ desired CLR level for NE under consideration], 

utilization factor to report — one each for CBR service. Spare capacity -[spa re capacity of NE under 

for nrtVBR service, and for the composite of all ser- consideration], 

vices 50 Utilization_factor=[utilization factor of NE under 

10) Determine Busy Period CLR Performance — ^ATM Pro- consideration]. 

tocol Processing with modifications corresponding to The result of this SELECT statement is a set of CAC values, 

those of above item 9. and PMOS 100 selects a CAC value based on that set of 

11) Determine CLR Performance Processing for Other CAC values. Various algorithms can be employed, such as 
Periods, with modification corresponding to those of 55 taking an average of the set of CAC values. 

above item 9. What is claimed is: 

1. A packet network that includes interconnected network 

Control of Adjustable Parameters of NEs elements, each of which has a connection admissions control 

Having obtained information from the various NEs in the (CAQ means and communication circuitry that reports on 

network and having made the various calculations and 60 performance of the network element relative to lost packets 

determinations disclosed above, PMOS 100 (or more and available capacity, the improvement comprising: 

particularly, tuner 170 therein) performs the analyses that a performance management operations system (PMOS) 

lead to the recommendations that go to console 300 and the controller that includes 

control signals that go to the NEs. In our embodiment, means for receiving signals from said network ele- 

PMOS 100 generates and sends the developed control 65 ments; 

signals periodically (every 24 hours) but, of course, there is an analysis module that develops a measure of busy 

no requirement for periodicity, or for any particular time periods of said network elements, traffic loads at 
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ports of said network elements during said busy 
periods, and utilization factors at said ports; and 
a transmission module that sends said measures devel- 
oped by said analysis module to a console, to affect 
operation of said network elements, 
where said busy periods corresponds to a selected "time- 
consistent" time interval or a "bouncing busy" time 
interval, where the time -consistent time interval is a 
selected interval of a first duration that is within an 
interval of second duration, and the bouncing busy time 
interval is an interval of said first duration that has more 
traffic than other intervals of first duration within said 
second internal. 

2. The network of claim 1 where said analysis module also 
develops load measures for virtual cotmections. 

3. The network of claim 1 where said analysis module also 
develops load measures for virtual paths. 

4. The network of claim 1 further comprising: 

a recommendations module that, based on measures 
developed by said analysis module, develops measures 
relating to specification of connection admission 
parameters in said network elements and outputs said 
measures developed by said recommendations module, 
to affect operation of said network elements. 

5. The network of claim of 1 where said analysis module 
develops measurements of said busy periods, said traffic 
loads, and said utilization factors for each of said network 
elements. 

6. A packet network that includes interconnected network 
elements, each of which has a connection admissions control 
(CAQ means and communication circuitry that reports on 
performance of the network element relative to lost packets 
and available capacity, the improvement comprising: 

a performance management operations system (PMOS) 
controller that includes 

means for receiving signals from said network ele- 
ments; 

an analysis module that develops a measure of busy 
periods of said network elements, traf&c loads at 
ports of said network elements during said busy 
periods, and utilization factors at said ports; 
a transmission module that sends said measures devel- 
oped by said analysis module to a console, to affect 
operation of said network elements; 
a recommendations module that, based on measures 
developed by said analysis module, develops measures 
relating to specification of connection admission 
parameters in said network elements and outputs said 
measures developed by said recommendations module, 
to affect operation of said network elements; 
where the measures developed by said recommendations 
module indicate whether said CAC means are to be 
instructed to make a connection admission threshold 
more stringent, less stringent, or left said connection 
admission threshold unaltered. 

7. The network of claim 4 where the measures developed 
by said recommendations specify threshold levels at which 
said CAC means should be set. 

8. A packet network that includes interconnected network 
elements, each of which has a connection admissions control 
(CAC) means and communication circuitry that reports on 
performance of the network element relative to lost packets 
and available capacity, the improvement comprising: 

a performance management operations system (PMOS) 
controller that includes 

ports for receiving signals from said network elements; 
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a processing module that develops a measure of busy 
periods of said network elements, a measure of 
capacities of said network elements, and a utilization 
factors of said network elements; 
a recommendations modiile that, based on measures 
developed by said processing module, develops mea- 
sures relating to specification of connection admis- 
sion parameters in said network elements; and 
a transmission module that sends said measures devel- 
oped by said recommendations module to affect 
operation of said network elements; where 
at least some of said network elements include an ability 

to provide diverse levels of Quality of Service, 
said network elements that support diverse levels of 
quality of Service provide information to said PMOS 
relative to trafQc that each of said network elements 
supports at the diverse levels of Quality of Service, and 
said recommendation module develops said measures 
relating to specification of connection admission 
parameters in said network elements so as to meet 
Quality of Service expectations of traffic carried by said 
network elements. 

9. The network of claim 8 where said network elements 
are of more than one type. 

10. The network of claim 8 where said network elements 
are manufactured by more than one manufacturer. 

11. The network of claim 8 where said processing module 
develops a measure of busy periods of said network 
elements, a measure of capacities of said network elements, 
and a utilization factors of said network elements for virtual 
connections of said network elements. 

12. The network of claim of 8 where said processing 
module develops a measure of said biisy periods, said 
capacities, and said utilization factors for each of said 
network elements. 

13. A packet network that includes interconnected net- 
work elements, each of which has a connection admissions 
control (CAC) means and communication circuitry that 
reports on performance of the network element relative to 
lost packets and available capacity, the improvement com- 
prising: 

a performance management operations system (PMOS) 
controller that includes 

ports for receiving signals from said network elements; 
a processing module that develops a measure of busy 
periods of said network elements, a measure of 
capacities of said network elements, and a utilization 
factors of said network elements; 
a recommendations modtile that, based on measures 
developed by said processing module, develops mea- 
sures relating to specification of connection admis- 
sion parameters in said network elements; and 
a transmission module that sends said measures devel- 
oped by said recommendations module to affect 
operation of said network elements, 
where the measm"es developed by said recommendations 
indicate whether said CAC means are to be instructed 
to make a connection admission threshold more 
stringent, less stringent, or left said connection admis- 
sion threshold unaltered. 

14. A packet network that includes interconnected net- 
work elements, each of which has a connection admissions 
control (CAC) means and communication circuitry that 
reports on performance of the network element relative to 
lost packets and available capacity, the improvement com- 
prising: 
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a performance management operations system (PMOS) 19. The network of claim 4 where said transmission 

controller that includes module sends said measures developed by said rccommen- 

ports for receiving signals from said network elements; dations module to said network elements. 

a processing module that develops a measure of busy 20. The network of claim 19 where said transmission 

periods of said network elements, a measure of 5 module sends said measures only to those of said network 

capacities of said network elements, and a utilization elements that need to have their connection admission 

factors of said network element; control thresholds altered. 

a recommendations module that, based on measures 21. The network of claim 19 where said transmission 

developed by said processing module, develops mea- ^^^^^ ^^^^^ measures to all of said network elements. 

sures relatmg to specification of connection admis- lo 22. The network of claim 19 where said transmission 

sion parameters m said network ekments; and ^^^j^^ ^^.^ ^^^^^^ periodically. 

a transmission module that sends said measures devel- 1 * 1 r 1 • t u • 1 

23. The packet network of claim 1 where said signals 

oped by said recommendations module to aJiect , , , .„ ^. t_ 

operation of said network elements. "^^'""^ ,f ™J ^^^^ °' 

,,,, _c followmg: cells received, cells transmitted, cells lost due to 

where the measures developed by said recommendations . n 1 4 j . .1 *-r *• 

, .ui 1 u-i- -ji^A^ congestion, cells lost due to protocol errors, utihzation 

specify threshold levels at which said CAC means „ . j 1 r . i_i 

should be set factors, spare capacity, and values of tunable parameters. 

Is'-nTe nelw^k of claim 14 further comprising a console ^4. The packet network of claun 23 where said tunable 

that is coupled to said transmission module of said PMOS. parameters are parameters that affect loss of cells. 

16. The network of claim 15 where said transmission 20 25. The packet network of claim 1 where said signals 
module sends said measures developed by said recommen- received by said PMOS from said network elements arrive 
dations module to said console, at a number of different time intervals. 

17. The network of claim 16 where said transmission 26. The packet network of claim 1 where at least some of 
module also sends to said console a history of said measures the signals received by said PMOS arrive in response to 
developed by said recommendations module. 25 polhng of at least some of said network elements by said 

18. The network of claim 17 where said transmission PMOS. 
module sends said history in response to a signal from said 

console that requests said history. ***** 
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